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11.2.5 Using the Law of Total Probability with Recursion 

A very useful technique in the analysis of Markov chains is using law of total probability. In fact, we 
have already used this when finding n-step transition probabilities. In this section, we will use this 
technique to find absorption probabilities, mean hitting times, and mean return times. We will 
introduce this technique by looking at an example. We will then provide the general formulas. You 
should try to understand the main idea here. This way, you do not need to memorize any formulas. 
Let's consider the Markov chain shown in Figure 11.12. 



The state transition matrix of this Markov chain is given by the following matrix. 


P = 


0 i o i 

0 0 0 1 

Before going any further, let's identify the classes in this Markov chain. 


0 0 
o I 


0 

0 


Example 11.9 

For the Markov chain given in Figure 11.12, answer the following questions: How many classes are 
there? For each class, mention if it is recurrent or transient. 

• Solution 

o There are three classes: Class 1 consists of one state, state 0 , which is a recurrent state. 
Class two consists of two states, states 1 and 2 , both of which are transient. Finally, 
class three consists of one state, state 3 , which is a recurrent state. 

Note that states 0 and 3 have the following property: once you enter those states, you 
never leave them. For this reason, we call them absorbing states. For our example here, 
there are two absorbing states. The process will eventually get absorbed in one of them. 
The first question that we would like to address deals with finding absorption 
probabilities. 
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Absorption Probabilities: 

Consider the Markov chain in Figure 11.12. Let's define < 2 * as the absorption probability in state 0 if 
we start from state i. More specifically, 

ao = P(absorption in 0 |Xq = 0), 
oi = P(absorption in 0 |Xq = 1), 
ct 2 = P(absorption in 0 |Xq = 2), 

03 = P(absorption in 0|Xo = 3). 

By the above definition, we have Oo = 1 and 03 = 0. To find the values of Oi and O 2 , we apply 
the law of total probability with recursion. The main idea is the following: if X n = i, then the next 
state will be X n + 1 = k with probability p ^. Thus, we can write 

a { = y ^CLkPik, fori = 0,1, 2 ,3 (11.6) 

k 

Solving the above equations will give us the values of 01 and 02 - More specifically, using Equation 
11 . 6 , we obtain 


Oq — Oo, 

1 2 

a l — 

1 1 

a 2 — 

O 3 = O 3 . 

We also know Oq = 1 and 03 = 0. Solving for 01 and 02 , we obtain 


1 



Let's now define bi as the absorption probability in state 3 if we start from state i. Since 
O* + bj = 1 , we conclude 

1 3 

b 0 = 0 , bi = -, b 2 = -, 63 = 1 . 


Nevertheless, for practice, let's find the 6 *'s directly. 


Example 11.10 
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Consider the Markov chain in Figure 11.12. Let's define 6 * as the absorption probability in state 3 if 
we start from state i. Use the above procedure to obtain bi for i = 0, 1, 2, 3. 


• Solution 

o From the definition of bi and the Markov chain graph, we have 6 q = 0 and 63 — 1. 
Writing Equation 11.6 for i = 1, 2, we obtain 


b i = -b 0 + -b 2 



b 2 = ^ 1 bi + 2 & 3 



Solving the above equations, we obtain 



Absorption Probabilities 

Consider a finite Markov chain { X n , Tl = 0,1,2,**-} with state space 

S = { 0 , 1 , 2 ,Suppose that all states are either absorbing or transient. Let l 6 S be an 
absorbing state. Define 

di = P(absorption in l\X$ = i), for all i G S. 

By the above definition, we have CLi = 1 , and aj = 0 if j is any other absorbing state. To find the 
unknown values of di' s, we can use the following equations 

a,i - Yaw, for ieS. 

k 

In general, a finite Markov chain might have several transient as well as several recurrent classes. As 
n increases, the chain will get absorbed in one of the recurrent classes and it will stay there forever. 
We can use the above procedure to find the probability that the chain will get absorbed in each of the 
recurrent classes. In particular, we can replace each recurrent class with one absorbing state. Then, 
the resulting chain consists of only transient and absorbing states. We can then follow the above 
procedure to find absorption probabilities. An example of this procedure is provided in the Solved 
Problems Section (See Problem 2 in Section 11.2.7) . 
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Mean Hitting Times: 


We now would like to study the expected time until the process hits a certain set of states for the first 
time. Again, consider the Markov chain in Figure 11.12. Let's define t j as the number of steps 
needed until the chain hits state 0 or state 3, given that Xq = i. In other words, ti is the expected 
time (number of steps) until the chain is absorbed in 0 or 3, given that Xq = i. By this definition, 
we have to = t% = 0. 


To find t\ and t 2 , we use the law of total probability with recursion as before. For example, if 
Xq = 1, then after one step, we have X\ = 0 or X\ = 2. Thus, we can write 

1 2 

t\ — 1 + —to + —t2 

2 

= 1 + 3 fe - 

Similarly, we can write 

*2 = 1 + 2*1 + 2^ 3 

= 1 + l h - 

Solving the above equations, we obtain 


Generally, let A C S be a set of states. The above procedure can be used to find the expected time 
until the chain first hits one of the states in the set A. 

Mean Hittin g Times 


Consider a finite Markov chain { X n , n = 0, 1, 2, • • •} with state space 
S = {0, 1, 2, • • • , r}. Let A C S be a set of states. Let T be the first time the chain visits a 
state in A. For all i 6 S, define 

ti = E[T\Xq = i\. 

By the above definition, we have tj =0, for all j 6 A. To find the unknown values of t i 's, we 
can use the following equations 

ti = 1 + ^2 t k pi k , for ieS-A. 

k 


Mean Return Times: 
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Another interesting random variable is the first return time. In particular, assuming the chain is in 
state l, we consider the expected time (number of steps) needed until the chain returns to state l. For 
example, consider a Markov chain for which Xq = 2. If the chain gets the values 


Xq — 2, X\ — 1, X 2 — 4, X 3 — 3, X 4 — 2, X 5 — 3, Xq — 2, X 7 — 3, 


then the first return to state 2 occurs at time Tl = 4. Thus, the first return time to state 2 is equal to 
4 for this example. Here, we are interested in the expected value of the first return time. In particular, 
assuming Xq = l, let's define 77 as the expected number of steps needed until the chain returns to 
state l. To make the definition more precise, let's define 

Ri = min{n > 1 : X n = l}. 

Then, 

n = E[R,\X 0 = i]. 

Note that by definition, Ri > 1, so we conclude Vi > 1. In fact, 97 = 1 if and only if l is an 
absorbing state (i.e., pu = 1). As before, we can apply the law of total probability to obtain Ti. 
Again, let's define tk as the expected time until the chain hits state l for the first time, given that 
Xq = k. We have already seen how to find tfi s (mean hitting times). Using the law of total 
probability, we can write 


n = 1 + ^ Plktk■ 
k 

Let's look at an example to see how we can find the mean return time. 


Example 11.11 

Consider the Markov chain shown in Figure 11.13. Let tk be the expected number of steps until the 
chain hits state 1 for the first time, given that Xq — k. Clearly, t\ = 0. Also, let r\ be the mean 
return time to state 1 . 

1. Find £2 and £ 3 . 

2. Find T\. 
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Figure 11.13 - A state transition diagram. 


• Solution 

o 1. To find ^2 an d ^3» we use the law of total probability with recursion as before. For 
example, if Xq = 2, then after one step, we have X\ = 2 or X\ = 3. Thus, 
we can write 

1 1 2 

^2 - 1 + -^2 + -^ 3 - 

Similarly, we can write 

*3 = 1 + 2*1 + 2 t2 

= i + \h. 

Solving the above equations, we obtain 

7 

*2 = 5, tz = 

2. To find 7 * 1 , we note that if Xq = 1, then X\ = 1 or Xi = 2. We can write 


r 1 = 1 + 9 

Xi 

• *1 + 7^2 

1 

1 

= i + - 

• 0 + — • 5 

2 

2 

7 


“ 2 ' 



Flere, we summarize the formulas for finding the mean return times. As we mentioned before, there 
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is no need to memorize these formulas once you understand how they are derived. 

Mean Return Times 

Consider a finite irreducible Markov chain { X n , n = 0, 1, 2, • • •} with state space 
S = {0, 1, 2, • • • , r}. Let l G S be a state. Let r/ be the mean return time to state l. Then 

n = l + 

k 

where tk is the expected time until the chain hits state l given Xq = k. Specifically, 

*1 = 0 , 

tie = 1 + tjPkj , for k^l. 

j 


previous 
next —> 
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